INTRODUCTION

The development in recent years that can be summarised as ‘more for less, faster and better’ has an impact on all IT disciplines.

With highly advanced development environments, developers can design and build complex programs relatively easily and quickly. The iterative development methods that are based on far-reaching interaction with the users ensure, among other things, that projects make interim deliveries faster. Such interim deliveries are then evaluated against the users’ wishes and requirements and the defects are reworked in the software. This means that the software changes continuously and regression risks are always there. Moreover, development is based more and more often on reusing internal and external components that must be integrated into the existing IT architectures. This has reduced the time required to develop new systems, putting testing even more emphatically on the critical path in terms of development and maintenance. It even threatens to become an obstructing factor.

All these factors, taken together with the fact that testing is already perceived to be a time-consuming and costly activity, make higher productivity of the tester and higher quality of the test a requirement. Test tools can be used as an instrument to achieve this.

Making test tools available to testers is often the responsibility of a separate department. One reason is the fact that setting up and maintaining test tools is a specific expertise. It is something of which testers generally have little knowledge. Another reason for making test tools the responsibility of a separate department is that big investments are often required to introduce tools in an organisation. In addition to the high acquisition costs, investment is required in training the people and developing new procedures. In other words, it takes time to realise a return on investment, often longer than one single project.

This section discusses test tools and their use in greater detail. Section “Test tools explained” below explains what a test tool is. Section Types Of Test Tools then describes the various types of test tools. Section “Advantages of using test tools” discusses the advantages of using test tools. The subsequent sections describe how test tools can be implemented in test organisations on the basis of a tool policy. To this end, Implement Test Tools With A Tool Policy (SP) explains the concept of tool policy and describes the life cycle model. The three phases are then listed, i.e. the Initiation, Realisation and Operation phases.

TEST TOOLS EXPLAINED

One of the conditions for the successful use of test tools is the existence of a structured test approach. In a properly controlled process, tools can certainly add a lot of value, but they are counter-productive in an inadequately controlled test process. In fact, test tools automate the test process, which requires a certain repeatability and standardisation in the activities to be automated. An unstructured process cannot comply with these conditions. The deployment of test tools, however, can serve to leverage the implementation of a structured approach. However, the least that is required is structuring and automation combined.

Terminology: tools, test tools and CAST tools

Tools used in a test process are referred to in different ways. For instance, some simply talk of tools, others of test tools, CAST tools (CAST stands for Computer Aided Software Testing), or test automation. It is not possible to make an unequivocal choice for the right terminology. There are parties that state that a tool is a test tool when it can be used exclusively to support a specific test activity. The counter-argument is that some test tools that serve to support test execution are sometimes used for other work. One example is a test tool that can be used to automate test execution. This tool works on the basis of automating operations and can also be used for data conversion. And that makes it a tool in a wider sense again. TMap uses the terms tools and test tools interchangeably.

Being able to use test tools is now assumed to be one of the tester’s basic skills. However, being able to set up and manage test tools and the far-reaching automation of routine work (e.g. test execution) still requires specialist and in-depth knowledge of programming and tools. Not every tester has
that knowledge. As a result, new types of specialism have emerged: test tool programmer, test tool expert, and test tool consultant.

Price structure of test tools

There are all kinds of test tools, all with their own price structure. Commercial tools often have a licensing system where a one-off price is agreed based on the number of users of the tool. In addition to this one-off price an annual contract is signed, ensuring the organisation that the tool’s supplier will provide support and new updates and releases. Often, this is called a maintenance or service contract.

In addition there are test tools with price structure on the basis of the variants shareware, freeware and open-source software. The price structure for shareware is such that it can be distributed without or with few restrictions, but a fixed price having to be paid when used repeatedly. Freeware is software for which the author has issued a licence for use and further distribution in unchanged form without requiring compensation. Open-source software goes one step further than freeware. In addition to the free distribution of the software, the author gives permission for modifying the software. The modified software can also be distributed freely. Contrary to opensource software, freeware is protected fully by copyright. And contrary to open-source software, the source code of freeware is not usually made available. More and more (self-made) test tools are made available by the creators through the Internet on the basis of these variants.

ADVANTAGES OF USING TEST TOOLS

A test tool introduces an automated instrument that provides support to one or more test activities. The introduction of test tools can have various advantages, which are described below. These advantages do not apply to all of the tool types discussed above. This is specified for each advantage.

Increased productivity

By using a test tool for routine test work, the tester has much more time for other tasks. In particular with tools for automated test execution, a sizeable quantity of tests can be executed ‘unmonitored’, e.g. at night. This means that much more thorough testing can be done in different fields, but also that the test environment can be used more efficiently. The test environment is then used 24 hours per day, with test execution at night and the results being verified during the day. All this means much more testing in the same time.

Overnight backups

Backups are usually made during the night. This is when the systems are not being used and most resources, such as CPU and memory, are available. When an automated test must be executed at night, it is important to determine in advance whether backups are made during that period. These might block the automated test execution, which will only be detected until the following morning. This also applies to interfaces and other systems that are shut down at night.

Tool types that manage data (e.g. testware management tools and defect management tools) take over the routine management tasks from the tester. The labour-intensive task of creating reports is also reduced to one click of a button when using these tools. This leaves more time for other activities.

Higher testing quality

The use of test tools to support a structured test approach is an emphatic next step towards higher testing quality and quality software. The reason is the consistent execution of an activity that is supported by the tool. A tool imposes a standard work method, eliminating the human factor. The human factor makes that not every operation is precisely the same as the other, increasing the risk of errors. This effect becomes stronger with drawn-out repetitive activities, such as writing similar physical test cases or the execution of the test scripts. This standard work method also means that all information (input and output) for an activity is presented in a consistent manner. This minimises interpretation errors by different testers in a team.

Very regularly, deviation from prescribed steps or operations results in finding defects. Therefore the human (unpredictable) factor is important. As such, deploying tools for automated testing is never the only and ultimate solution to realise a high-quality test.

More work enjoyment

The execution of routine tasks is experienced as boring. When such tasks are automated, it increases the work enjoyment of the test team in addition to increasing reliability. Naturally this also results in improved productivity of the test team. Moreover, test teams generally perceive working with tools as pleasant. The reasons are the professional image towards the rest of the organisation and the new tasks (realisation and maintenance of the tools) that are created due to the use of the test tools.

Extension of test options

Some tests cannot be simulated fully when done manually. One example is the execution of stress tests. The deployment of test tools is virtually indispensable here. The deployment of a large number of ‘real’ users to achieve representative manning is usually feasible once, with a lot of effort, but not twice. Moreover these tools can trace defects that are very difficult to detect manually. For instance, measuring memory use of a website when more than 1000 visitors are online. Another example is testing a system without a user interface (e.g. middleware). Such a system can be accessed by using a test tool.